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Claims 1-11 have been examined, claims 1-7, and 1 1 are rejected, and claims 8-9 
are objected to, but allowable if rewritten in independent form. 

The Examiner objected to the abstract of the disclosure. The abstract has been 
amended to overcome the objection. 

The Examiner objected to claim 6 regarding the relative size and relationship of the 
rectangle and the polygon. The claim clearly states that the display area is enclosed 
by the polygon. That the display area is a rectangle, and that the rectangle is in fact 
a largest possible rectangle that can be enclosed by the polygon, see Figure 1, 135- 
136. However, to expedite the examination, claim 6 has been rephrased to make 
the relationship even more definite. 

The Examiner objected to claim 8 regarding maximum likelihood estimation. 
Maximum likelihood estimation (MLE) has been well known in the art for a couple 
of centuries. The principles of MLE can be traced back to Pierre Laplace (1749- 
1 827), also see Scott R. Eliason, "Maximum Likelihood Estimation," Series in 
Quantitative AppHcation in the Social Sciences, Sage University Papers, 1996, 
Nico Nagelkerke, "Maximum Likelihood Estimation of functional relationships," 
Springer Verlag, January 1992, Press, W. H.; Flannery, B. P.; Teukolsky, S. A.; 
and Vetterling, W. T. "Robust Estimation." §15.7 in Numerical Recipes in 
FORTRAN: The Art of Scientific Computing, 2nd ed. Cambridge, England: 
Cambridge University Press, pp. 694-700, 1992. 

Formally, an estimation technique which is insensitive to small departures from the 
idealized assumptions which have been used to optimize the technique. Also see 
any of the following patents, 6,374,216 Penalized maximum likelihood estimation 
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methods, the baum welch algorithm and diagonal balancing of symmetric matrices 
for the training of acoustic models in speech recognition, 6,097,776 Maximum 
likelihood estimation of symbol offset, 5,604,724 Maximum likelihood estimation 
using reference and spontaneous output peaks of partial response equalizer, 
5,355,431 Text Signal detection apparatus including maximum likelihood 
estimation and noise suppression, 5,296,861 Method and apparatus for maximum 
likelihood estimation direct integer search in differential carrier phase attitude 
determination systems, 4,959,715 Method and apparatus for converting an 
interlaced television transmission to a sequential display using maximum 
likelihood estimation, 4,684,955 Maximum likelihood estimation of G/T of 
satellite earth terminals using extraterrestrial radio sources, 4,422,165 Maximum 
likelihood estimation of the ratio of the velocities of compressional and shear 
waves. 

Applicant takes note that maximum likelihood estimation is well known to those of 
ordinary skill in the art. Certainly, there is no need for the Applicant to explain 
well known mathematical principles that date back centuries. 

The invention projects in turn, for each one of a plurality of projectors, a single 
registration image onto a display surface. A union of the single registration images 
forms a polygon. An input image is acquired in turn, for each registration image by 
a camera. A single projective matrix is determined between the display area and 
each input image for each projector. 

In other words, the projective matrix is determined entirely and directly only from 
the single projected registration images and nothing else. 
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Source images for each projector are warped according to the single projective 
matrices, and pixels of the warped source images are weighted according to the 
single projective matrix. The warped and weighted source images are then 
projected concurrently and directly onto the display surface to form a mosaic 
image. 

The warping according to the invention is performed using a homography that is 
determined during a single calibration step using a single registration image. 

It should be noted that the single projective matrix as claimed is a homography. As 
formally defined in the art, a homography is a parametric function that maps pixels 
in a first image to pixels in another image. The homograph can be expressed 
functionally as (x,y) = H(u,v), where the function H is in the form of a 3x3 matrix. 
As an advantage, any pixel location can be mapped simply by applying the 
function H. A homography does not require a table look-up, nor interpolation. For 
further details on the formal requirements for a homography see U.S. Patent No. 
5,598,515 issued to Shashua on January 28, 1997, and U. S. Patent No. 6,527,395 
issued to Raskar et al. on March 4, 2003 

The Examiner rejects claims 1-7, and 1 1 under 36 U.S.C. 102(b) as being 
anticipated by Surati (U.S. 6,456,339). 

In contrast to the invention, Surati uses a two step indirect calibration process with 
two test charts, see Figures 6 and 7 and the corresponding text in the specification. 

To establish a screen to camera mapping (SC) a first test chart (43) is placed on the 
screen and an image is taken of the test chart, see Figure 7. 
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To establish a projector to camera mapping (PC) a second test chart (71) is 
projected on the screen and an image is taken of the second test chart, see Figure 8. 

The two calibration images are than used to establish a non-parametric mapping 
between the projector and the screen that is used to warp images, see Figure 3 and 
4A. 

The non-parametric mapping of Surati uses a 2D look-up table (315), see column 
1 1, lines 15-20, "The outcome of the calibration phase 301 is a two-dimensional 
lookup table 315. During the display phase 303, the lookup table 315 is used by an 
image warper 317 to warp an image 319 to be displayed." Also see the look-up 
mapping in Figures 1 lA-1 IC. 

One of ordinary skill in the art would not confuse the discrete look-up table of 
Surati with the continuous functional projective matrix as claimed. 

Obviously, as a disadvantage, a look-up table requires interpolation for discrete 
pixels that are not in the table, see column 16, lines 56-67. 

As an advantage, the parametric projective matrix according to the invention is 
continuous over all pixels. 

In short, Surati is distinguished and inferior in two counts. One, the calibration is a 
two step process requiring two test charts and two images per projector, and the 
mapping used for warping is discrete and non-parametric. 

The claimed method uses a single calibration step with a single registration image, 
and the warping is done is parametric according to a projective matrix in the form 
of a homography. 
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Claims 2-4 deal with a display surface that is oblique to an optical axis of the 
projectors. Column 8, lines 26-38 make no reference to: an oblique angle, nor an 
optical axis. 

The test chart 71 in Surati is not a checkerboard pattern. One of ordinary skill in 
the art would not recognize the test chart on the left below with a checkerboard 
shown on the right. There are no black squares in the test chart. There is not an 
equal number of altemating black and white squares. There is space between the 
squares. 




Moreover the features 72 are landmarks that need to be carefully spaced. "The 
space 74 between landmarks presents an additional concern. A large space between 
landmarks tends to reduce the effect light from adjacent landmarks has on a 
particular landmarks' measurement. On the other hand, a small space between 
landmarks tends to reduce the number of required test chart pictures by having as 
many landmarks as possible on one screen. In experiments, the landmarks are 
separated by 30 pixels, eliminating the measurable effect from adjacent landmarks, 
yet limiting the total number of required test screen pictures to sixteen per 
projector," column 13, line 1 1 et seq. 

Landmarks are precluded in checkerboards. 

Surati describe a display area that is a largest rectangle enclosed by a polygon. 
Figure 1 does show a single rectangle. Nowhere does Surati describe 
homographies. His relationships are table look-ups. 
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All rejections have been complied with, and appHcant respectfully submits that the 
application is now in condition for allowance. The applicant urges the Examiner to 
contact the applicant's attorney at phone and address indicated below if assistance 
is required to move the present application to allowance. Please charge any 
shortages in fees in connection with this filing to Deposit Account 50-0749. 




Respectfully submitted, 
MitsubishiElectric Research Laboratory Inc. 



Andrew jrCurtin 
Reg. No. 48,485 
Attorney for Assignee 



Mitsubishi Electric Research Laboratories, Inc. 
201 Broadway 
Cambridge MA, 02139 
(617) 621-7573 
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S/N: 09/930,425 

Specification 

Please amend the abstract as follows. 

Abstract of the Disclosure 

A method forms a mosaic image on a display surface with a multiple projectors. 
For each projector in turn, a registration image is projected onto the display surface 
so that a union of the projected registration images forms a polygon. With a 
camera, for each registration image in turn, a corresponding input image is 
acquired. A display area on the display surface enclosed by the polygon is then 
identified, and a single projective matrix between the display area and each input 
image is determined for each projector. A source image for each projector is 
warped according to the corresponding homography of the projector. The pixels of 
the warped source image are weighted according to the single projective matrix, 
and then the warped and weighted source images are concurr e tnly concurrently 
projected directly onto the display surface to form the mosaic image. 
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